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Abstract 

The notion of a Radon transform is introduced for completely integrable billiard tables. In 
the case of Liouville billiard tables of dimension 3 we prove that the Radon transform is 
one-to-one on the space of continuous functions K on the boundary which are invariant 
with respect to the corresponding group of symmetries. We prove also that the frequency 
map associated with a class of Liouville billiard tables is non-degenerate. This allows us to 
obtain spectral rigidity of the corresponding Laplace-Beltrami operator with Robin boundary 
conditions. 

1 Introduction 

This paper is concerned with the integral geometry and the spectral rigidity of Liouville billiard 
tables. By a billiard table we mean a smooth compact connected Riemannian manifold {X,g) 
of dimension n > 2 with a non-empty boundary T := dX. The elastic reflection of geodesies 
at r determines continuous curves on X called billiard trajectories as well as a discontinuous 
dynamical system on T*X - the ^^billiard flow" - that generalizes the geodesic flow on closed 
manifolds without boundary. The billiard flow on T*X induces a discrete dynamical system in 
the open coball bundle B*T of F given by the corresponding billiard ball map B and its iterates. 
The map B is defined in an open subset of B*T = G T*T : \\S,\\g < 1}, where \\C\\g denotes 
the norm induced by the Riemannian metric g on the corresponding cotangent plane and it 
can be considered as a discrete Lagrangian systems as in [9], [11], [15]. The orbits of B can 
be obtained by a variational principal and they can be viewed as "discrete geodesies" of the 
corresponding Lagrangian. In this context, periodic orbits of B can be considered as "discrete 
closed geodesies" . 

Let ^ he a. positive continuous function on B*T. Denote by tt^K the pull-back of the 
continuous function K £ C{T) with respect to the projection Trp : T*T T. We are interested 
in the following problems. 

Problem A. Let X be a continuous function on T such that the mean value of the product 
TTpX • /X is zero on any periodic orbit of the billiard ball map B. Does it imply K = 0? 

The mapping assigning to any periodic orbit 7 = {^O; ■ ■ • > 0m-i} C B*r of the map B 
the mean value (1/m) Ejlo (^f ^ ' A*) iOj) of the function tt^K • // on 7 can be viewed as a 
discrete analogue of the Radon transform, considering the periodic orbits of the billiard ball 
map as discrete closed geodesies. Problem A has a positive answer for any ball in the Euclidean 
space M" centered at the origin if /i = 1 and K is even. In fact, approximating the great circles 
on the sphere by closed billiard trajectories of the billiard table we obtain from the hypothesis 
in Problem A that the integral of K over any great circle is zero. Since K is even, by Funk's 
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theorem we obtain K = {\^, Theorem 4.53]). The case of general Riemannian manifold is 
much more complicated. 

Denote by ttx : T*X ^ X the natural projection of the cotangent bundle T*X onto X. Let 
S*X\r = G T*X : irxiO £ MWg — 1} be the restriction of the unit co-sphere bundle to 
r. There are two natural choices for the function /i we are concerned with, namely, = 1 or 
yu(^) = (7r+(^), 71^)^-'^, ^ G B*T, where (•, •) is the standard pairing between vectors and covectors, 
Ug is the inward unit normal to T at x = 7rr(0i tt'^ : B*T ^ S* X\r assigns to any ^ e T^F 
with norm \\(,\\g < 1 the unit outgoing covector the restriction of which to T^F coincides with 
^. Recall that a covector based on x is outgoing if its value on ng{x) is non-negative. The 
latter choice of /x is related with the wave-trace formula for manifolds with boundary obtained 
by V. Guillemin and R. Melrose [4l[5|. It appears also in the iso-spectral invariants of the Robin 
boundary problem for the Laplace-Beltrami operator obtained in ^12j. From now on we fix the 
positive function fi G C{B*T) by 

Ax^l, orby ^^{0 = {^^^{^),ngr\ i^B*T. (1.1) 

For that choice of //, it will be shown that Problem A has a positive solution for a class of 
Liouville billiard tables of classical type. A Liouville billiard table (shortly L.B.T.) of dimension 
n > 2, is a completely integrable billiard table (X, g) (the notion of complete integrability will be 
recalled in Sect.[2|) admitting n functionally independent and Poisson commuting integrals of the 
billiard flow on T*X which are quadratic forms in the momentum. A L.B.T. can be viewed as a 
2"""^ -folded branched covering of a disk-like domain in M" by the cylinder T"~^ x [—N, N], where 
T = R/Z and > 0. Liouville billiard tables of dimension two are defined in [10] and in any 
dimension n > 2 in [11], where the integrability of the billiard ball map is shown via the geodesic 
equivalence principal. Here we write explicitly first integrals of the billiard flow and show that 
it is completely integrable (see Sect. 13. ip . An important subclass of L.B.T.s are the Liouville 
billiard tables of classical type having an additional symmetry and for which the boundary is 
strictly geodesically convex (with respect to the outward normal —ng). It turns out that the 
group of isometries of a L.B.T. of classical type is isomorphic to (Z/2Z)". Moreover, the group 
of isometries of {X,g) induces a group of isometries G on F which is isomorphic to (Z/2Z)"'. An 
important example of a L.B.T. of classical type is the interior of the n-axial ellipsoid equipped 
with the Euclidean metric. More generally, there is a non-trivial two-parameter family of L.B.T.s 
of classical type of constant scalar curvature k having the same broken geodesies (considered as 
non-parameterized curves) as the ellipsoid [11', Theorem 3]. This family includes the ellipsoid 
(k = 0), a L.B.T. on the sphere (k = 1) and a L.B.T. in the hyperbolic space (k = —1). 

Theorem 1. Let {X,g), dimX = 3, be an analytic L.B.T. of classical type. Suppose that there 
is at least one non-periodic geodesic on the boundary F. Choose fj, as in Let K G C(F) he 

invariant with respect to the group of isometries G = (Z/2Z)'^ of the boundary F and such that 
the mean value of ir^K ■ /i on any periodic orbit of the billiard ball map is zero. Then K = 0. 

In particular. Problem A has a positive solution for ellipsoidal billiard tables in with /i = 1 as 
well as for //(^) = (7r"'"(^), n^)"^, for any K £ C(F) which is invariant under the reflections with 
respect to the coordinate planes Oxy, Oyz, and Oxz- More generally. Theorem 3 can be applied 
for any L.B.T. of the family described in [111 Theorem 3]. The condition that the boundary 
contains at least one non-closed geodesic will become clear after the discussion of Problem C. 
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As it was mentioned above the map assigning to each periodic orbit of the bilhard ball map B 
the mean value of ir^K • /i on it can be considered as a discrete analogue of the Radon transform. 
Another version of the Radon transform can be defined as follows. Denote by J- the family of 
all Lagrangian tori A C B*T which are invariant with respect to some exponent B"^, m > 1, of 
the billiard ball map B, i.e. i?™(A) C A. For any continuous function on F we denote by 
TZk,h{^) the mean value of the integral of tt^K • /i on A G with respect to the Leray form (see 
Sect. [2]). The mapping A i— > TZK,iii^), A G .F, will be called a Radon transform of K as well. 

Problem B. Let K he a continuous function on T which is invariant with respect to the group 
of isometries G. Does the relation TZk,^ = imply K = 0? 

The main result of the paper is the following theorem, which gives a positive answer of Problem 
B for L.B.T.s. 

Theorem 2. Let {X,g), dimX = 3, be a Liouville billiard table of classical type. Fix /i by 
jll.l]) . If K G C(r) is invariant under the group of symmetries GofT and'R.K,^i{^) = for any 
A G JP", then K = 0. 

We point out that L.B.T.s of classical type are smooth by construction but they are not supposed 
to be analytic. 

A similar result has been obtained for the ellipse in [4] and more generally for L.B.T.s of 
classical type in dimension n = 2 in jpLOj and [L2i- It is always interesting to find a smaller set 
of data A for which the Radon transform is one-to-one. In the case n = 2 the proof is done by 
analyticity, and we need to know the values of the Radon transform TZk,^{^) only on a family 
of invariant circles {Ajjjgi^ approaching the boundary S*T of B*T. The case n = 3 is more 
complicated, since the argument using analyticity does not work any more. Nevertheless, we can 
restrict the Radon transform to data "close" to the boundary in the following sense: It will be 
shown in Sect. 13.31 that any L.B.T. of classical type of dimension 3 admits four not necessarily 
connected charts Uj, 1 < j < 4, of action-angle variables in B*T. Two of them, say Ui and 
U2, have the property that any unparameterized geodesic in S*T can be obtained as a limit of 
orbits of B lying either in Ui or in U2 (then the corresponding broken geodesies approximate 
geodesies of the boundary). Moreover, in any connected component of Ui and U2 there is such a 
sequence of orbits of B, while and C/4 do not enjoy this property. In other words, the charts 
Ui and U2 can be characterized by the property that there is a family of "whispering gallery 
rays" issuing from any of their connected components. For this reason the two cases j = 1,2 
will be referred as to boundary cases. Denote by J^b the set of all A G J'-" lying either in Ui or 
in U2 ■ We will show in Theorem 14.11 that the restriction of the Radon transform 'R-k,^! on J^h 
determines uniquely K. 

As an application we prove spectral rigidity of the Robin boundary problem for Liouville 
billiard tables. Given a real- valued function K G C(F,M), we consider the "positive" Laplace- 
Beltrami operator A on X with domain 




where H'^{X) is the Sobolev space, and ng{x), x G F, is the inward unit normal to F with respect 
to the metric g. We denote this operator by ^g,K- It is a selfadjoint operator in Lp'{X) with 
discrete spectrum 



Spec Ag^i^ := {Ai < A2 < • • • } , 
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where each eigenvalue A = Xj is repeated according to its multiphcity, and it solves the spectral 
problem 

Au = Xu in X , 
an , , (1.2) 



dug 



lr = K u\r ■ 



Let [0, 1] B t Kf & C°°(r,IR) be a continuous family of smooth real- valued functions on F. 
To simplify the notations we denote by At the corresponding operators Ag^Xf This family is 
said to be isospectral if 

VtG [0,1], Spec (At) = Spec(Ao) . (1.3) 

We consider here a weaker notion of isospectrality which has been introduced in [12j. Fix two 
positive constants c and d > 1/2, and consider the union of infinitely many disjoint intervals 

(Hi) X := [ofc, hk\, Q < ai < hi <■■■< < bj. <■■ ■ , such that 

lim Ofc = lim = +oo , lim (6^ — a^) = , and a^+i — b^ > cb^'^ for any k > 1 . 

fe— >00 fc— >00 k—rOO 

We impose the following "weak isospectral assumption" : 

(H2) There is a > such that Vt G [0, 1] , Spec (A^) n [a, +00) C X, where X is given by (Hi). 



Using the asymptotics of the eigenvalues Xj as j — > 00 we have shown in [12] that the condition 
(Hi)-(H2) is "natural" for any d > n/2 {n = dimX), which means that the usual isospectral 
assumption (11. 3p implies (Hi)-(H2) for any such d and any c > 0. 

Theorem 3. Let {X,g) be a 3- dimensional analytic Liouville billiard table of classical type such 
that the boundary T has at least one non-periodic geodesic. Let 

[0,1] 3t^ Kte C7°°(r,R) 

be a continuous family of real-valued functions on T satisfying the isospectral condition (Hi)- 
(H2). Suppose that Kq and Ki are invariant with respect to the group of symmetries G = (Z/2Z)^ 
o/r. Then Kq = Ki. 



A similar result has been proved in [12] for smooth 2-dimensional billiard tables. The idea of 
the proof of Theorem [3] is as follows. Fix the continuous function ^ by /i(^) = (7r"'"(^), n^)"^. 
First, using [12^ Theorem 1.1] we obtain that 

1Zk,A^)=1Zk,A^), (1.4) 

for any Liouville torus A of a frequency vector satisfying a suitable Diophantine condition. Next, 
we prove that the union of such tori is dense in the union of the two charts C/j, j = 1, 2, of "action- 
angle" coordinates in B*T, which implies ()1.4|) for any torus A £ J^t,. Now the claim follows 
from Theorem 14. 1[ In the same way we prove Theorem [TJ First we obtain that 7^x^^(A) = for 
a set of "rational tori" A. Then we prove that the union of these tori is dense in Ui U U2, and 
we apply Theorem 14. 1[ We point out that the proof of Theorem [3] presented in Sect. [6] requires 
only finite smoothness of Kt (see Theorem 16. ip . 

An important ingredient in the proof of both theorems is the density of the corresponding 
families of invariant tori in Uj, j = 1,2. This follows from the non-degeneracy of the frequency 
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map for Liouville billiard tables of classical type studied in Sect.O Recall that in any chart 
Uj of action-angles coordinates the frequency map assigns to any value of the momentum map 
the frequency vector of the minimal power B"^ : Uj ^ Uj, m > 1, that leaves invariant the 
corresponding Liouville tori A C Uj. The frequency map is said to be non-degenerate in Uj if 
its Hessian with respect to the action variables is non-degenerate in a dense subset of Uj. We 
are interested in the following problem: 

Problem C. Is the frequency map non-degenerate in any chart of action-angle coordinates? 

We prove in Theorem 15 . 1 1 that this is true in the charts Uj, j = 1, 2, for any analytic L.B. T. 
of classical type for which the boundary T admits at least one non-closed geodesic. The 3-axial 
ellipsoid and more generally any billiard table of the two-parameter family of L.B.T.s of classical 
type of constant scalar curvature described in [TTl Theorem 3] has these properties. 

The non-degeneracy of the frequency map appears also as a hypothesis in the Kolmogorov- 
Arnold-Moser theorem. In particular, Theorem 15.11 allows us to apply the KAM theorem for the 
billiard ball maps associated with small perturbations of the L.B.T.s in [11^ Theorem 3]. It is a 
difficult problem to prove that the frequency map of a specific completely integrable system is 
non-degenerate. The non-degeneracy of the frequency map of completely integrable Hamiltonian 
systems has been systematically investigated in j7]. The main idea in [7] is to investigate the 
system at the singularities of the momentum map. In our case we reduce the system at the 
boundary S*T of B*T. To our best knowledge this problem has not been rigorously studied for 
completely integrable billiard tables even in the case of the billiard table associated with the 
interior of the ellipsoid. 

The article is organized as follows. In Sect. [2] we recall certain facts about the billiard ball 
map and define a Radon transform for completely integrable billiard tables. Sect. [3] is concerned 
with the construction of L.B.T.s. First we consider a cylinder C = T^^ x T^^ x [—N, N], where 
Ti = M/K for / > and > and define a "metric" g and two Poisson commuting quadratic 
with respect to the impulses integrals /i and I2 of g in C. The non-negative quadratic form 
g is degenerate at a submanifold 5 of C To make g a Riemannian metric we consider its 
push-forward on the quotient o" : C — > (7 of C with respect to the group generated by two 
commuting involutions ai and CJ2 whose fix point set is just S. The main result in this section 
is Proposition 13.31 which provides C with a differentiable structure such that the push- forwards 
g := CF-tg, Ii := (T*7i and I2 := c7*/2 are smooth forms, 5 is a Riemannian metric on C and Ii and 
I2 are Poisson commuting integrals of g. In Sect. 13.31 we write an explicit parameterization of 
the regular tori by means of the values of the momentum map corresponding to the integrals Ii 
and I2. The injectivity of the Radon transform is investigated in Sect. HI The non-degeneracy 
of the frequency map of an analytic L.B.T. is investigated in Sect. [5j The proof of Theorem 
[T] and Theorem [3] is given in Sect. El In the Appendix we investigate the frequency map and 
the action-angle coordinates of completely integrable billiard tables and derive a formula for the 
frequency vectors of B^. 

2 Invariant manifolds, Leray form, and Radon transform 

In the present section we define the Radon transform for integrable billiard tables. First we recall 
the definition of the billiard ball map B associated to a billiard table {X,g), dimX = n, with 
boundary F. Denote hy H & (7°°(T*X, M) the Hamiltonian corresponding to the Riemannian 
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metric g on X via the Legendre transformation and set 

S*X := {e G T*X : H{0 = 1} , S*X\r := G S*X : 7rx(e) G T} , 

5iX|r := U G 5*^|r : ±{(,ng) > 0} , 

Ug being the inward unit normal to T. Denote by r : T*X\y' — > T*X\y the "reflection" at 
the boundary given hy r : v i-^ w, where w\Tyr = v\Tyr and {w,ng) + {v,ng) = 0. Obviously 
r : S*X\r S*X\r. Take u G 5'!f.X|r C r*X and consider the integral curve 7(t;n) of the 
Hamiltonian vector field Xh on T*X starting at u. If it intersects transversally S*X\y at a time 
ti > and lies entirely in the interior of S*X for t G (0, ii), we set Bo{u) := 'y{ti,u) G S'l^Xlr- 
The set O C S^Xlr of all such u is open in S'^Xlr- The billiard ball map is defined by 

Denote hy B*T := {(^ ^ T*T : H(S^) < 1} the (open) coball bundle of T. The natural projection 
7r_(_ : 5!j_X|r — > -B*r assigning to each u G S*X\r the covector ii|Ta;r G B*T admits a smooth 
inverse map vr^ : i?*r — > S'^XIt- The map i? := vr+oiJovr^ is defined in the open subset 7r+(0) 
of the coball bundle of T and it is a smooth symplectic map, i.e. it preserves the canonical 
symplectic two-form uj = dp A dx on B*T. The map B will be called a billiard ball map as well. 

From now on we assume that the billiard ball map B : B*T ^ B*T is globally defined 
and completely integrable. By definitior0, the complete integrability of the billiard ball map of 
{X,g) means that there exist n — 1 invariant with respect to B smooth functions Fi, ...,Fn-i on 
B*T which are functionally independent and in involution with respect to the canonical Poisson 
bracket on T*T, i.e. 

{F„Fj} = 0, l<i,j<n-l. 

The functions Fi, Fn-i are said to be functionally independent in B*T if the form dFi A ... A 
dFn-i does not vanish almost everywhere. A function / on B*T is said to be invariant with 
respect to the billiard ball map B \i B* f = f . The invariant functions with respect to the 
billiard ball map are called also integrals. In particular, as i^i, ...^Fn-i are integrals, then any 
non-empty level set 

L, ■.= {ieB*T ■ = ci,...,F,„i(e) = c„_i}, c = (ci,...,c„_i) gM"-\ 

is invariant with respect to the billiard ball map B : B*T B*T. By Arnold-Liouville theorem 
any regular compact component Ac of Lc is diffeomorphic to the {n — l)-dimensional torus 
T"~^ and there exists a tubular neighborhood of Ac in B*T symplectically diffeomorphic to 
W^~^ X T"'~^ that is supplied with the canonical symplectic structure Yl^=i d-Jk A dOk- Here 
Bp-i := {J = (Ji,..., J„_i) G M"-i : |J| < r} for some r > 0, 6 = (6*1, 6'„_i) are the periodic 
coordinates on T"""^, and | • | is the Euclidean norm in M"^^. The coordinates {J, 9) are called 
action-angle coordinates of the billiard ball map. Recall that Ac is regular if the (n — l)-form 
dFi A ... A dFn-i does not vanish at the points of Ac. Any regular torus Ac is a Lagrangian 
submanifold of B*T and it is also called a Liouville torus. 

Assume that the Liouville torus Ac is invariant with respect to B'^ for some m > 1, i.e. 
B"^{Ac) = Ac. Let Oc be a (n — l)-form defined in a tubular neighborhood of Ac in B*T so that 

w""^ := w A ... A a; = Qc A dFi A ... A dFn-i . (2.1) 
^This is one of the many definitions of complete integrability of billiard ball map. 
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It follows from (|2.1|) that the restriction Ac := ad Ac of Q^c to Ac is uniquely defined. The form 
Ac is a volume form on Ac which is called Leray form. As B preserves both the symplectic 
structure uj and the functions Fi, one obtains from ()2.ip that the restriction of B"^ to 

Ac preserves Ac- 

Fix a positive continuous function /i on B*T and denote by J- the set of all Liouville tori. 
For any continuous function X on F the mapping TZk,^ : given by 

T^K,f. (Ac) := j-^ (vrJK)^ Ac , (2.2) 

is called a Radon transform of K. It is easy to see that the Radon transform does not depend 
on the different choices made in the definition of the Leray form. 

Remark 2.1. An alternative definition of the Radon transform would he 

nK,A^c) := YJJT) E / iB*y{i^rKh)>^c (2.3) 

where m > 1 is the minimal power of B that leaves Ac invariant, i.e., B"^{Ac) = Ac- Note that 
(|2.3p appears as a spectral invariant of (II. 2p in fl^ - We show in Sect. that for L.B.T. of 
classical type m = 1 in the charts Ui and 112- In particular, (|2.2p and (|2.3p coincide in this case. 

There is another notion of complete integrability which is related to the "billiard flow" of 
the billiard table {X,g) (cf. Definition 17. 2p . We reformulate Definition 17.21 in terms of the 
cotangent bundle T*X: A billiard table is completely integrable if there exist n smooth functions 
Hi, Hn-i, Hn = H va. a, neighborhood U of S*X in T*X with the following properties: 

(i) the functions Hj are in involution in U with respect to the canonical Poisson bracket on 
T*X, i.e. {HuHj]=Q, 1 <i,j <n, 

(ii) Hi, ...,Hn are functionally independent in U, 

(iii) r*Hj = Hj in U\r for I < j < n. 

The properties (i) and (iii) imply that Hj is invariant with respect to the billiard flow in U for 
any 1 < j < n. In particular, the functions Fj = Hj o Tr"*", 1 < j < n — 1, are integrals of the 
billiard ball map B. As Hi,...,Hn are functionally independent in U the billiard ball map is 
completely integrable if, for example, the integrals Hj are homogeneous functions with respect 
to the standard action of M* := M \ on the fibers of T*X \ 0. In this way we see that the 
billiard ball map of a completely integrable billiard table is completely integrable if the integrals 
are homogeneous functions on the fibers of T*X \ 0. 

Definition 2.2. A billiard table {X,g) with a completely integrable billiard ball map will be 
called 7^-rigid with respect to the density /x if Problem B has a positive solution. 
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3 Liouville billiard tables 



3.1 Construction of Liouville billiard tables 

In this section we describe a class of 3-dimensional completely integrable billiard tables called 
Liouville billiard tables. The interior of an ellipsoid is a particular case of a Liouville billiard 
table - see § 13.21 below as well as § 5.3 in [11] for the general construction of Liouville billiard 
tables of arbitrary dimension, where the integrability of the billiard ball map was deduced 
from geodesically equivalence principle. Here we write explicitly integrals of the billiard flow 
of a Liouville billiard table which are quadratic forms in momenta, and hence, homogeneous 
functions of degree 2 on the fibers T*X \ 0. 

For any > and any a;^ > {k = 1,2) consider the cylinder 

C ■.= {{61,92,63)} ^T^, xT^, X [-N,N], Ti :=R/IZ, 

where 61 and 62 are periodic coordinates with minimal periods loi and 102 respectively and ^3 
takes its values in the closed interval [— A^, A^]. Define the involutions ai,a2 ■ C ^ C of the 
cylinder C by 

ai:{6i,62,63)^i-6i,'^-62,6s) (3.1) 

and 

a2: {61,62,63) ^{61,-62,-63). (3.2) 

As the commutator [o"i, 172] vanishes one can define the action of the Abelian group A := Z2©Z2 
on C hy {a,6) ^ a ■ 6 := {a^^ o a2^){6), where a = (01,02) G A and 6 G C. Consider the 
equivalence relation on C defined as follows: The points p,q £ C are equivalent p q iff 
they belong to the same orbit of A (i.e., there is a € ^ such that a • p = q). Denote by C the 
topological quotient C/ ~^ of C with respect to the action of A and let 

a:C^C (3.3) 

be the corresponding projection. A point p £ C is called a regular point of the projection (j3.3p 
iff it is not a fixed point of the action for any ^ a € A. The points in C that are not regular 
will be called singular or branched points of the projection a. The set of singular points is given 
by S := SiU S2, where 

5i:={(ei^0(mod^),^2^ Y (™°'^t)'^3) : -N < 63 < 

and 

52 := {(^1, 02=0 (mod ^^), ^3 = 0) : 61 € T^,] . 

The set Si C C has four connected components homeomorphic to the unit interval [0, 1] C M 
while S2 <Z C has two connected components homeomorphic to T. 

Lemma 3.1. The space C is homeomorphic to the unit disk in M^. The map a : C ^ C is 
a A-folded branched covering of C . 

Remark 3.2. The image Si of Si under the projection a : C ^ C is homeomorphic to the 
disjoint union of two unit intervals and the image S2 of S2 is homeomorphic to T. 
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Proof of Lemma First consider the action of the involution ai on the cyhnder 

C = T^, xT^, X [-N,N]. 
For any value c G [—N, N] the involution cji is acting on the 2-torus := T^, x T<^, x {6*3 = c} 

by 

<Ti(c) :(^i,02)^ (-ei,Y-02). 

The involution cri(c) : — > has four fixed points and it is easy to see that the topological 
quotient S^^c) of with respect to the orbits of the action of cti(c) is homeomorphic to the 
2-sphere — {x G : jxp = 1}. Hence, 

Ci := (C/ - {(S'(c),c) : cG[-N,N]}. (3.4) 

Under the identification (j3.4p . the involution (T2 : Ci — > Ci becomes 

§2 X [-N, N] ^ §2 ^ ^] 

(xi,X2,X3;c) (xi,X2, -X3; -c) . 



The fixed points of this involution form a submanifold, {(xi,X2,0;0) : + = 1} = T, and 



the corresponding quotient is homeomorphic to B^, hence, C = {C\l ^a-^) — B^. □ 



In what follows we will define a differential structure T> on C and a smooth Riemannian met- 
rics g on the manifold X := {C,T)) = such that the billiard table {X,g) becomes completely 
integrable. The branched covering o" : C — > C defined above will play an important role in our 
construction. To this end choose three real- valued C°°-smooth functions ipi,'^2 : M — > M and 
</?3 : [— A^, A^] ^ R satisfying the following properties: 

{Ai) ipk {k = 1,2,3) is an even function depending only on the variable 9k and 
^i{Oi) > MO2) > > (^3(^3); 
ipk {k = 1,2) is periodic with period (j^; 

ip2 satisfies the additional symmetry 992(6*2) = ~ ^2^1 

(^2) '■= niin(/?/fc = max(/7/fc_|_i {k = 1, 2) and ui> U2 = 0; 
for any k G {1, 2}, ipk{6k) = i^fc iff 6*^ = mod ; 

'^2(6*2) = z^i iff 6*2 = ^ mod ; 
(/J3(e3) = ^2 iff ^3 = 0; 

(^3) compatibility conditions: 

(1) for k G {1,2}, = > and ^f\o) = V?\u;i/2)E 

(2) for any / > 0, ff^\o) = (-l)Vf ^ (^2/4) and ipf\o) = (-l)Vf ^(0). 



^Item (^45) (i) in [TT] has to be written similarly. 
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Consider the following quadratic forms on TC (quadratic on any fiber TqC) 

dg^ -.= 111(161 + 112(161 + 11^(101 (3.5) 

and 

dll := i^2 + ^3)'i^id6l + {ipi + ip3)U2de^ + i^i + ip2)Ilsdel 

(3.6) 

dll := {^2^3)^1 del + {^2^3)^2 ddl + (V32¥'3)n3 d6l 

where 

Hi := {^i - ^2){^i - ^3), := ((/?! - ^2){^2 - ^3), and Ilg := {ipi - ip^){(p2 - ^3)- 

We say also that the forms above are quadratic forms on C. Notice that dg^ is degenerate, it 
vanishes on S. 

Proposition 3.3. Assume that the functions ipi, (p2, ^3 satisfy (Ai) + (A3). Then there exists 
a differential structure D on C such that the projection a : C ^ C is smooth and a is a local 
diffeomorphism in the regular points. The push-forwards g := a^g and ■= (J*Ik {k = 1,2) 
are smooth quadratic forms and g is a Riemannian metric on X := {C,T>). In addition, the 
billiard table {X,g) is completely integrable and the quadratic forms Ii, I2, and :=^(^, 0/^; 
considered as functions on TX are functionally independent and Poisson commuting integrals 
of the billiard flow of g. 

Proof of Proposition Consider the set 5 = 5i U ^2 C C of branched points of the covering 
o" : C — > (7. Take a point p = {O^, 62, 6^) S and assume for example that 6^ = 0, 6^ = ^ and 
^3 G [—N,N]. Define a new chart Vi = {{xi,X2,xs)} in a neighborhood of p by x^ ■= 6^ — 6^, 
k = 1,2, and X3 := ^3, where \xk\ < for k = 1,2 and |x3| < A^. In this chart p = (0,0,^3) 

and 

dg'^ = Qidxj + Q2dxl + Q^dxl, (3.7) 

dif = {^2 + 03)Qi dxl + {^i + 03)Q2 dxl + (01 + ^2)Q3 dxl (3.8) 

dl! = (0203)Qi dxj + (0i03)Q2 dxl + (0102)^3 dxj, (3.9) 

where Qi := ((/>i - 02)(</'i - h), Q2 ■= (</>! - <p2){4>2 - ^3), Q3 ■= {4>i - (t>3){(t>2 - 4>3), (t>k{xk) = 
^ki&k + ^k), ^ = 1,2, and 03(2:3) = ip3{xs)- Note that Vi is a tubular neighborhood of the 
chosen component of Si and it does not intersect the other components of S. It follows from 
(Ai) + (A^) that the functions 0i, 02, and 03 are smooth and have the following properties in 
Vv 

(Li) 0fc is even and depends only on the variable Xk; 

{L2) 01 > 03 and 02 > 03; 

(-L3) 01 and 02 satisfy: 

(i) 01 > i/i if xi / and 0i(O) = vi, 0;'(O) > 0; 
{a) 02 < 1^1 if X2 / and 02(0) = vi, 02(0) < 0; 

^The canonical symplectic structure on T* X induces a symplectic structure on TX by identifying vectors and 
covectors by means of tfie Riemannian metric g. 
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(m) (/)f'^(0) = (-1)V?^(0) for any / > 0. 

In the new coordinates, the involutfon cri\vi becomes o"i|v\ : {xi,X2,X3) (— xi, — X2, X3). In 
order to define a differential structure in a neighborhood of cr{p) in C consider the mapping 
$1 : Vi ^ Im<I>i := Wi, 

$1 : (xi, X2, X3) 1-^ (yi = xj- xl, y2 = 2xiX2, yz = x^). (3.10) 

By Lemma [33] below the push-forwards g\wi '■= ^i*{g\vi)i h\wi '■= ^i*ih\vi), and /2IW1: = 
^i*(-^2|yi) are smooth quadratic forms on Wi and g\wi is positive definite. Since $1 o ((Ti|yj) = 
$1 and (T2{Vi) n Vi = we can identify a\vi with and get a differential structure in the 
neighborhood of (t{p) £ C. In a similar way we construct a tubular neighborhood V2 of the 
component 6'° = W2/2, 6*2 = ^ and 9^ € [-A^, A^] of Si together with a mapping ^2 ■ V2 ^ W2 
such that the push- forward of g\v2, Ii\v2i and I2\v2 smooth quadratic forms on W2. Consider 
also the tubular neighborhoods V3 := ct2(Vi) and V4 := (T2{V2) of the other two components of 
5*1 in C together with the mappings 

$3 := <I>1 o (fjalvs) : V'a ^ M^i and := ^2 ° [cr2\vi) -Vi ^ W2 . (3.11) 

For j = 3,4 one has <I>j o (cti|v,.) = <I>j, and therefore we can identify with (y\vj- As the 
quadratic forms ()3.5p and ()3.6p are invariant with respect to 02 we obtain from (13. lip that 

^3*{9\Vi) = fflvKi, ^4*(9|v'4) = 9\W2^ ^?.*{Ij\v-i) = i^Wi, and ^i^{Ij\vi) = Ij\w2, J = 1>2. In 
particular, the mappings a\vs : V3 Wi and a\v4, '■ V4 W2 and the push-forward of ()3.5p and 
(j3.6p with respect to them are smooth. Arguing similarly we treat the case p £ S2 and construct 
a coordinate chart W3 of S2 = cr{S2) in C. 

Covering the image of the branched points of a by the charts Wi, W2, and W3 we get a 
differential structure on uj^-^Wj D SiU 82- As the set C \ {Si U S2) consists of regular points 
of a we can induce a differential structure on it from the differential stricture of the cylinder C. 
The union of these two differential structures is compatible and defines a differential structure 
D on C. Denote by X the smooth manifold X = (C,P). It follows from (Ai) that the forms 
(j3.5p and (|3.6p on C are invariant under the involutions (13. ip and (|3.2p . In particular, the push- 
forwards g := £7*5, Ii := (T*Ii, and /2 := o"=k/2 are smooth quadratic forms on X \ {Si U 5*2). 
Moreover, we have seen that the push-forwards g, Ii, and I2 are smooth quadratic forms on Wj, 
and that 5 is a Riemannian metric in Wj for any j G {1,2,3}. Hence, the push-forwards g, Ii, 
and I2 are smooth quadratic forms on X and ^ is a Riemannian metric. We will show that Ii 
and I2 are integrals of the billiard flow of the metric g on C\S. Indeed, applying the Legendre 
transformation = 11^. 0^, k = 1,2,3 (which is well defined only on C \ S) and dropping for 
simplicity the factor ^ in the Hamiltonian we get 

h = ^Pl + ^d + ^pi (3.12) 
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which can be rewritten in Stdkel form (cf. [13], |13t §2]) 

' pI = ipj H - ifi h + h 

< pI = -^IH + ip2h- h ■ (3.13) 

In particular, the functions H, Ii, and I2 Poisson commute with respect to the canonical sym- 
plectic form uj := dpi A dOi + dp2 A d62 + dps A dOs on the cotangent bundle T*{C \ S) (see for 
example \13\ Proposition 1]). Moreover, the forms Ii and I2 are invariant with respect to the 
reflection map at the boundary p : {TC)\qc — > {TC)\qc given by 

{61,62, ±N, 61,62, 63) ^ (61,62, ±N, 61,62, -63) . 

Hence Ii and I2 are Poisson commuting integrals of the billiard flow of the metric g on X\ (t{S). 
As cr{S) is a 1-dimensional submanifold in the 3-manifold X we get that Ii and I2 are Poisson 
commuting integrals of the billiard flow of the metric g. A direct computation shows that H, Ii 
and I2 in (|3.12p are functionally independent on T*{C\S). Hence, H, Ii and I2 are functionally 
independent on r*(X \ o-(5)). □ 



Lemma 3.4 . The quadratic forms g\wi = ^i*(fl'|vi); h\wi = ^i*(-^i|yi)) o.nd hlwi 
<I>i^(/2|vi) '^'"s smooth and g\wi positive definite. 

Proof of Lemma \3.4\ A direct computation involving (jS.lOp shows that 

'^^^l(vi/i\Si) = 9u dyf + 2^12 dyidy2 + 522 dyi + 533 dyj 

where 



1 / (Ai - 02 \ ( {(Pi- + ih - h)x2 '\ ~ _ 1/ 01 -4> 2\^ 



and 

522 



l[-^J[ J' ^33 = (01-03)(02-03). (3.15) 



Let A := 0ig|^ (g) dxi + 4>2-£^ ® dx2 + <Pi^ O dxs £ C°°(TVi T*Vi). A similar computation 
as above shows that 

~ d ~ d ~ d d 

where 

r 4)1X1+4)2X1 ~ 01-02 7 01 - 02 7 0ix|+02xf 
^11 = 2~ 2 ' ^12 = 2 ^1^2 , ^21 = 2 ^1^2 , ^22 = 2 ' (3-16) 
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Consider the tensor field A as a section in Horn {T*Vi, T*Vi). Then we have 

det{A + c)g{iA + c)-'C, = c^9{i, + chiC, + HC, (3.17) 
for any c > — max ipi(9i). We will show that the coefficients (|3.14|) . (j3.15|) . and (|3.16p . when 

0<6i<ti;i 

re-expressed in terms of the variables (yi, 2/2) Us), are smooth in Wi. Then the statement of the 
Lemma will follow from the relation (j3.17p and the properties of the Vandermonde determinant. 
Consider, for example, the function 

^{xi,X2):= 2~ — 2 ' (2;i,X2) 7^ (0,0) . 

1 2 

Fix m G N, m > 1. Using (L3) and the Taylor formula with an integral reminder term we get 

m 

Mxi) = ^Ofexf +xj"+^5i,2m+l(xi) 

k=0 
m 

MX2) = 5](-l)'^afcxf +X2'"+152,2™+1(X2) 
k=0 

where Sj^2m+i, j = 1, 2, are smooth functions in a neighborhood of 0. Lemma [331 below implies 
that 

m—l 

^{xi,X2) = ^k{yi,y2) + S2m+l{xi.,X2) for (xi , X2) / (0, 0) , 

fc=0 

where $^(2/1, 2/2) := -Pfc(2/i, 2/2) for A:-odd and $^(2/1, 2/2) := Rkivi, Vi) for A;-even are homogeneous 
polynomials of degree 2k with respect to (2/1,2/2), and 

C , , X^+^gi,2r.+l(xi) - X^'"+^52,2m+l(x2) . 

S2m+l{Xi,X2) := — ■ 5 ■ , (xi,X2) / (0,0). 

1 ~^ 2 

Consider the directional derivatives 

1 ( \ A pi — ^ — ^ ( ^ 

''^Wi^ 2(xf + xl) V^d^i ~ ''^8^2) ''^W2^ 2(xf + xl) + 9^ 

We have 

lim dy^dyo S2m+l{xi,X2) = 

for a + (3 < m. Hence, <I> can be extended by continuity to a C°°-smooth function in the 
variables (2/1,2/2) in a neighborhood of (0, 0) and its Taylor series is Yl'kLo ^^(2/1,^2)- In the case 
when 01 and (j)2 are real analytic the power series 'Yl'k'=Q^k{yi-,y2) is uniformly convergent in a 
neighborhood of (0,0). 

Arguing similarly we obtain that the coefficients (|3.14p - (j3.16p are C°°-smooth in the variables 
(2/1, y2) when (pi and 02 are smooth and real analytic if (pi and 02 are real analytic. Moreover, 
by Taylor's formula 0i(a;i) = vi + aix\ + o{x\) as xi — > and 02(^2) = — aix\ + o(x|) as 
X2 — > that together with (j3.14p and (j3.15p implies gu = ai{vi — 03) + o(l), gi2 = o(l), and 
922 = o-ii^i — 03) + 0(1) as 2/ ^ ((0,0,2/3)). Hence, '^5^I(vKi\5i) extended by continuity 

to (0,0,2/3) ^ '^1 ^'^'i by (-^2) the extension is positive definite. This completes the proof of the 
Lemma. □ 
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Lemma 3.5. For any m > 2, 



2m 2m 
Xi — X2 



ixl + xl)Pm-i{yi,y2), m-even, 
Qm{yi,y2), m-odd, 



^2m I 2m 



{xl + xj) Rm~i{yi,y2), m-odd, 
Nmiyi,y2), m-even, 
where Pm, Qm, Rm, o-iT'd are polynomials of yi and y2 of degree m. 

Proof of Lemma \3.5\. Introduce the complex variables z := xi + 1x2 and w := yi + iy2 and note 
that w = z^. Then, for any m > 2, xf*" ± xl^ = (^{z + zf"^ ± {-l)'^{z - z)^™) /T^"^. Finally, 
using Newton's binomial formula one concludes the Lemma. □ 

Following [TU] we impose the following additional assumptions on the functions (p^: 
(^4) (/Pi(^i) = (/7i(wi/2-^i) 

(^5) for any k G {1,2} the derivative '^'^{Ok) > on (0,a;fc/4) and if'-^iO^) < on (0,iV]. 

The condition ip'^{N) < means that the boundary of X is locally geodesically convex. 

Definition 3.6. The billiard table (X, g) in Proposition 13.31 is called a Liouville billiard table 
(shortly L.B.T.). Liouville billiard tables satisfying conditions (^4) and {A^) are called Liouville 
billiard tables of classical type. In the case when (pi, ip2, and ip^ are real analytic, the billiard 
table is called analytic L.B.T. 

The involutions, 

(01,02,^3) ^ (-^1,^2,^3 

(^1,^2,^3) ^ (^-^1,^2,^3) (3.18) 



2 

?l,^2,^3) ^ (^1,-^2,^3) 



induce a group of isometrics G{X) = G{X, g) on X which is isomorphic to the direct sum 

G{X) ^ Z2 © Z2 © Z2 . 



Remark 3.7. The action of G{X) on {X,g) is an analog of the action of the group Z2©Z2©^2 
in the interior of the ellipsoid in = {{x, y, z)} generated by the reflections with respect to the 
coordinate planes Oxy, Oyz and Oxz- 

(21) (21) 

Remark 3.8. The compatibility conditions (p\ (0) = cpi {uJi/2), / = 0,1,..., in (A3) follows 
from {A4) for L.B.T.s of classical type. 
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3.2 Ellipsoidal billiard tables 



Denote by the Euclidean space 



{{xi, X2, X3)} supplied with the standard Euclidean 



metric dgQ 



dxf + + dx^. A class of L.B.T.s in M depending on 3 real parameters 



61 > 62 > ^3 can be obtained using the mapping: 



xi 

X2 
X3 



A2) 



A, 



63 



cos (pi 



63 - Ai 



63 sin (pi cos 02 
'3 sin (j)2 



hi 



are periodic 
gives a 4- 



where Afc := hk+i + (bk - bk+i) sin"^ (pk (k = 1,2), A3 := 63 - (/>a: (/c = 1,2 
coordinates with period 2tt, and —N < (ps < N. The mapping Eq : x [—N, N] — 
folded branched covering of an ellipsoidal domain X in M'^ and {X, qq) is a L.B.T. of classical type 
- for details see § 5 in [11]. More generally, the two-parameter family of billiard tables (M^, g'^^^) 
of constant scalar curvature k, in [111 Theorem 3] consists of L.B.T.s of classical type according 
to § 5.4 in [11]. The boundary of any billiard table of the family is geodesically equivalent to the 
ellipsoid. In particular, it has non-periodic geodesies and satisfies the hypothesis of Theorem 
[Hand Theorem [3l This family contains the ellipsoid (k = 0) and L.B.T.s of both positive and 
negative scalar curvature that are realized on the standard sphere and on the hyperbolic space 
respectively. 



3.3 Parameterization of the Lagrangian tori 

The aim of this section is to obtain charts of action-angle coordinates for L.B.T.s of classical type 
and to parameterize the corresponding Liouville tori. Recall that a L.B.T. {X,g) is obtained as 
a quotient space of the cylinder 

C = {{ei (modcui),02 (modu;2),^3)} = T^i x T^^ x [-N,N] 

with respect to the group action of ^ = Z2 © Z2 as described in Sect. 13.11 By Proposition 
13.31 the projection a : C ^ X \s smooth and invariant with respect to the group action of A 
on C. Moreover, the push-forwards of the quadratic forms ()3.6p with respect to the projection 
cj : C — > X are integrals of the billiard flow on {X, g). The boundary dC of C has two connected 
components defined by ^3 = itA^ and we set 

:= {{01 (modcJi), 62 (modu;2), ^3 = N)}. 

By construction the restriction crl^j^ of the projection a : C — > X to is a double branched 
covering of the boundary T = dX. 

Denote Cr := C \ S and introduce on T*C the coordinates ^2, ^3;Pi,P2,P3)}, where 
Pi, P2, and p3 are the conjugated impulses. The Legendre transformation corresponding to the 
Lagrangian Lg{£^) := 5(^,0/2, ? £ TCr, transforms the Lagrangian and the integrals (13. 6p to 
the functions H, h and h on T*Cr given by (l3T2]) FI Set 

Qi := r*a|T^ = {V = {0,p) G T*Cr : ^3 = N}. 
*For simplicity we drop the factor i in the Hamiltonian function. 
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The restriction tDi of the symplectic two-form 

uj = dpi ^ dOi + dp2 A d62 + dps A d^s 



to Qi is Cji := wIqj = dpi A d9i + dp2 A d62- This form is degenerate and its kernel Kerwi is 

d 
dps ■ 

Q := {7? G T*a : H{rj) = 1} 



spanned on the vector field jr-- Denote by Q the isoenergy surface 



and consider the set Q2 ■= Q ri Qi. It is clear that Q2 is diffeomorphic to the restriction of the 
3gCr of Cr to the torus T^. 



unit cosphere bundle S*Cr of Cr to the torus T?r. The set 



Q+ :={r^ = {e,p)eQ2 : P3 < 0} 

can be identified with the set S'^Crlj^^^ of all rj in 5*6*^1x2^ such that {r],ng) > 0, where n 
denotes the inwar 
be identified with 



gt^rlT^ sucii LiiaL \ii,ng/ ^ u, wiieie /t^ 
^ \ ^1. iviuicuvci, Ulic open uuinaic ^g\i^j^ 



denotes the inward unit normal to \ 5i. Moreover, the open coball bundle i?*(T^ \ ^i) can 



{(^i,^2;Pi,P2)gT*(t2\5') : + ^) |e3=iv < 1} , (3.19) 



where S' := {{0i = (mod^),02 = x (mod^f )}. Consider the map R : B*g{T% \ Si) ^ Q 
given by 



+ 



R : {9i,e2;pi,P2) ^ {Oi,02;Pi,P2,P3) where ps = -^/UsJl - ^ - ^ . 

The coball bundle -B*(T^ \ "S*!) can be considered as a phase space of the billiard ball map 
B : B*{T\ a{S)) B*{T\ a{S)) via the branched double covering cj|t2^ : T^^ ^ T. In this 
setting the map R can be identified with 7r+. We have also 0J2 ■= R*oji = dpi A d9i + dp2 A d92. 
Moreover, the functions Ii := R*Ii and I2 '■= R*l2 are functionally independent integrals of B 
mB*g{T%\Si). 

In the coordinates {{Oi,02;pi,P2)} the integrals Ii and I2 become (cf. p.l2p ) 

2 2 

11 = i^i + V>2) - {^1 - ^3)^ - {^2 - '^s)^ , (3.20) 

ill 112 

12 = (pi^2 - ^2{^l - 2^3)77- - ^l{^2 - 1^3)77" ' (3-21) 

111 i-i-2 

where := ip^^N) < z^2 = in view of (^1) and (^2)- In order to describe the invariant 
manifolds of the billiard ball map B we choose real constants hi and /i2 and consider the level 
set 

Lh := {Ii = hi,l2 = h2} C B;{T% \ Si), h = {hi, h2). 
Consider the quadratic polynomial, 

P{t) := f - hit + h2 = {t- Ki){t - K2) , (3.22) 
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where ki and K2 are the roots of P and hi = ki + K2, /i2 = kiK2. If (01,02', Pi 1P2) G -^fci it 
follows from (|3.13p that 

= ipliOi) - hiifiiOi) + h2=pl>0, (3.23) 

- ^(¥^2(^2)) = -¥^2(^2) + hiM02) -h2=pl>0, (3.24) 

and 

P{ip3{N)) = P{us) = vl - hius + h2>0. (3.25) 

Then the set Lh is non-empty if and only if there is a point (^1,^2) G (M./uJi'Z) x (M/a;2Z) such 
that the inequalities (j3.23p . (j3.24p . and (j3.25p are satisfied. In particular, it follows from (j3.23p 
and (I3.24P that the roots ki < K2 are real, hence, D := h\ — 4/i2 > 0. Moreover, {Ai) {A2) 
imply 

vi < Vi{Oi) < z^o , 
< = 1^2 < M02) < jyi , (3.26) 
1^3 = 953(A^) < 1^2 = 0. 
Then the following four cases can occur: 

(A) t's < Ki < z^2 = and = 1^2 ^ 1^2 ^ 1^1', 

(B) f3 < Ki < z^2 = and 1^1 < K2 < vq] 

(C) = z^2 < < ^2 < z^i; 

(D) = 1^2 ^ ^1 ^ 1^1 and z^i < ^2 < fQ. 

Consider the union JJi of all Li^ in i?*(T^ \ Si) such that (A) with strict inequalities holds for 
the corresponding (ki, K2)- We will see below that any Lh in Ui is a disjoint union of Liouville 
tori. In the same way we define U2 corresponding to (B), C/3 corresponding to (C) and C/4 
corresponding to (D). Denote Uj := a^{Uj) C B*r, j = 1,2,3,4, where is the push-forward 
of covectors corresponding to a : C ^ X. 

Definition 3.9. We refer to cases (A) and (B) as to boundary cases and denote J^i, := UiUU2- 

Remark 3.10. We will see in 5ect.O that the billiard trajectories in T*X issuing from Ui U U2 
"approximate" the geodesies on the boundary T. 

We are going to parameterize the invariant tori belonging to the level set Lh. To that end we need 
the inverse functions of </'i|[o,lji/4] and v?2|[o,w2/4]- According to (^i) (A^) the function ipi has 
the following properties. It is a periodic function of period a'i/2, ipi{ui/A + 61) = ipi{uJi/A — 9i) 
for any 9i, the map 921 : [0,a;i/4] — > [i'i,i'q] is a homeomorphism, Lp'i{9i) > in the interval 
(0,u;i/4), and the critical points of ipi at = and 9i = coi/A are non degenerate. Denote by 
fi : [ui^uq] — > [0,u;i/4] the inverse map of '/3i|[o,a;i/4]- Then fi is smooth in (z/i,fo), /( > in 
that interval, and 

fi{xi) = F^{^xi - vi) as xi^vi+Q, 

(3.27) 

fi{xi) = F{{^uq - xi) as 
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where are smooth functions in a neighborhood of 0, and 
F+(0) = 0, Ff(0)=u;i/4, (F+)'(0) = ^2ip'l{0)-^ and (Ff )'(0) = -2ip'l{u;,/A)-\ (3.28) 

The function (/32|[o,a;2/4] same properties, and we denote by /2 : [0, i^i] — > [0,1^2/4] its 

inverse function. Then /2 is smooth in (0, z^i) and > in that interval, and 

/■ 

f2ix2) = F^{,/x^) as X2^0 + 0, 

(3.29) 

/2(x2) = F^i^/i^i - X2) as X2^i^i-0, 

V 

where are smooth functions in a neighborhood of and 

F2+(0) = 0, F,-{0) = 0^2/4, {F^nO) = ^2^'2'(0)-^ (i^2")'(0) = -^-2(^'2'(^2/4)-i. (3.30) 

Assume that Lh C C/i. We have < ki < and < K2 < i^i. It follows from (j3.23p - (|3.24p and 
(j3.26p that Lh consists of four connected components T^'^^ (1 < A; < 4) which are diffeomorphic 
to T^. Moreover, the image of each T^'^^ with respect to the bundle projection T*T^ — > 
coincides with one of the annuli 

A'h := {0 < 01 < UOi; -f2{K2) <02< f2{^2)} 

and 

4 := {0 < 01 < c^i; 1^2/2 - f2{^2) < 02 < W2/2 + /2(k2)} . 

Assume that the tori T^"^ and are projected onto Aj^ and similarly, T^^'* and T'ff^ are 
projected onto vl'^. As the map clf^^ • '^at ^ T is invariant with respect to the involution 

^ : (01,02) ^(-0i,c^2/2- 02), 

and «(j4'^) = ^4^, the pairs {T'j^\ Tjf'^) and (T^^\ T^^^) correspond the same pair of invariant tori 
in r*r, which we identify with (T^^^T^^^). It follows from ([323]), (IgTM]) and (13:26]) that the 
map r^jej 

: A'^ T^^^ defined by, 

(01,02)'^ (01, 02; eWMOi? - hiMOi) + h2, £2 V-'/52(02)2 + hip2{02) " 112) , (3.31) 

gives a parametrization of the torus T^^^ for ei = 1 and £2 = ±1. In the same way, taking 

(2) 

ei = — 1 and £2 = ±1 we parametrize . 

In the same way one treats the cases (B), (C) and (D). In particular, one gets that Ui, U2, 
and C/3 have 4 connected components while 6^4 has 8 connected components. Similarly, Ui, U2, 
and have 2 connected components and U4 has 4 connected components. 

4 7^-rigidity 

We are going to prove that Liouville billiard tables of classical type are 7?--rigid with respect to 
the densities n defined by (jl.ip . 
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Theorem 4.1. Let {X,g) be a Liouville billiard table of classical type and let K G C(r,]R) be 
invariant with respect to the action of the group G{X) on V . Suppose that TZk,ia.{^) = for any 
Liouville torus A G J-^- Then K = 0. 



Proof of Theorem \4-l\ First, consider the case when = 1. Denote the pull-back of K under 
the projection o'\j2^ : — > F by /C, /C G C(T^). Let Ah G ^6 be a Liouville torus and let 
Th be a connected component of {a\j2^)*Ah C B*{T'j^) where h = {hi, /12) are the values of the 
integrals Ii and I2 on T^- Then we have 



{Xhin})-^ [ ICXh = 2nK,i(Ah) = 0, 



where Xh is the corresponding Leray's form on T^- Note that /C is invariant under the involution 
(^1,^2) ^ —02) since a\j2 is invariant under the involution (j3.ip for 6^ = N. Recall 

that that the group G{X) = Z2 © Z2 © defined by (j3.18p acts by isometrics on X and on 
its boundary F. Since K is invariant under this action, the function /C(^i,^2) is invariant with 
respect to the involutions 

(^1,^2) ^(-^1,^2), (^1,^2)^(^^1/2-^1,^2) (4.1) 

and 

(^1,^2)^(^1,-^2), (^1,^2) ^(^1,0^2/2 -02). (4.2) 

From now on we consider /C G C(T^,M) which is invariant with respect to the involutions ()4.ip 
and ()4.2p and such that for any 

/ ICXh = yTh^H. (4.3) 

First, take h = {hi, /12) and assume, for example, that T^ C Ui. We shall give an explicit formula 
for the Leray form on the connected components of Lh, using the parameterization obtained in 
Sect. [331 Set Th := T^^\ and let r+ be the "half torus" rii(A^), where the map ru is defined 
by (|3.3ip . Consider the set 

A'h{6) := {0 < 01 < ioi; -f{K2) + 5 < 62 < f{K2) - 5} , 

where 5 > is sufficiently small. It follows from (j3.3ip that the functions {9i, 62,^1,12) give a 
coordinate chart in a neighborhood of the branch T^{5) := rii(A^(5)) C Tj^ . We will compute 
the Leray form on it. In the coordinates {{0i,92',Pi,P2)} on B*T'j^ we have 

002 A 0)2 = 2 dpi A dOi A dp2 A d62 



= 2 d{y/MOiy - Ti^i{Oi) + 12) A ddi A d{^-^2{e2? + T1MO2) - T2) A d92 

= {M0.)-M02))d0iAd02 . 

2 ^(^1(01)2 (01 ) +J2V-¥P2 (02)2 +Xl(^2 (02) -X2 

In particular, letting 5 ^ + we see that the Leray form on can be identified with 

^ {M0i)-MO2))deiAde2 

■ VMOl)^ - hlMOl) + /i2 V-^2(02)2 + /ll(^2(02) - h2 ' 
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We have 



ICXh 



f 2(1^2) fi^i 
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f'^-^^r^/^ ic{ei,e2){MGi) - ^2(92)) deid92 

Jo y^i<pi{9i) - Ki){ipi{ei) - K2)y^{ip2i02) - Ki)(k2 - (p2{02)) ' 



as the functions JC, (pi, and ip2 are invariant with respect to the involutions (j4.ip and (j4.2p . Set 

K {91,82) := }C{9i,92){MSi) - and denote 

Jo Jo ^/{^l{9l) - Ki){ipi{9i) - K2)^/{(p2{92) - ki){k2 - ^2(92)) 



Then ()4.3p imphes 

Ma(ki,k2) =0 
for any ki G (—2^3, 0) and any K2 G (0,z^i). 

Remark 4.2. A/^oie that for any fixed H2 G (0, ui) the function ki — > Ma{hi , H2) can he extended 
to an analytic (possibly multivalued) function on C\ ([0,^2] U [^^ij^'o])- Since it vanishes for 
Ki G (— t's, 0) we obtain that Ma{k2, ki) = 0, Vki G C \ ([0, K2] U [ui, z/q]) and \/k2 G (0, z/i). 

Set 

Kl(xi,X2) :=^(/l(xi),/2(x2))/((xi)/^(x2). 

It follows from (13:271) and ([3:29]) that Ki G (1/1,1/0) x (0,i/i)). More precisely, pl^f]) and 
([3:29]) imply 

Lemma 4.3. We have 

i> ( N K{fi{xi)j2{x2))F{xi,X2) , . 

Ki{xi,X2) = (4.6) 

^/Xl - Vi^VQ - Xx^fxi^Vx - X2 

where the function [x\,X2) ^ -f^(/i(a^i), /2(2:2)) continuous on [1/1,1/0] x [0,i/i], the function 
F G C([z/i,z/o] X [0,z/i]) does noi dependent on K and F > 0. 

Passing to the variables xi = 921(6*1) and X2 = 922 (6*2) in (|4.5p we get 

, . . . r°r^ i^i(xi,x2) dx2dxx ^ , . 

Ma{ki,K2)= / w Y ^^ ^ = ° ^^-^^ 

i;/iiO V(^l ~ '^l)(2;2 - Ki)y^{xi - K2){K2 - X2) 

for any ki G (— cxd,0) and any K2 G (0,z/i). Consider now the case {B). Arguing in the same 
way we obtain 

16 M JO V(xi - ki)(x2 - Ki)i/(xi - «;2)(k2 - a:;2) 

for any ki G (— oo,0) and K2 G (z/i,z/o). 
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In the same way one obtains: 
Case (C): For any < ki < K2 < ui, 



Mc{ki,K2) 



12 fl^O 



Ki{xi,X2) dxidx2 



'n JkJui a/ {Xl - Ki){x2 - Kl)^/{Xl - K2){k2 - X2) 

Case (D): For any ki £ {0,ui) and K2 G (z^i,i/o), 

I'"'/'"" Ki{xi,X2) dxidx2 



ICXh 



a/ (Xi - Ki){x2 - {Xi - K2){k2 - X2) 



(4.9) 



(4.10) 



Remark 4.4. In what follows we will not use the identities ()4.9p and (I4.10p . 

Now, we argue as follows: Take a continuous function x on the interval [0, u\\ and consider the 
mean 



Ma(ki,K2)x(k2) C?/«2 







where Ma(ki,K2) is given by (|4.7p . In view of Lemma 14.31 we can apply Fubini's theorem to 
the following integral 







Ma{^', Kl) 



K2 rVQ 



Ki{xi,X2)x{i^2) dxidX2 



Jui v^(xi - Kl)(x2 - Kl)y^{xi - K2){k2 - X2) 



dKo 



KlixuX2) 



Jui-s/ixi - Kl){x2 - Ki) 



X{k2) dK2 



•i\J{xx - K2){k2 - X2) 



dxidx2 



(4.11) 



for any ki G (— oo,0) and any x G C([0,i^i])- Similarly, consider the mean 



1^0 



MB{f;Ki):= / Mb{ki,K2)x{i^2) dK2 



where x is a continuous function on the interval [i^i, We obtain as above 

-ynr^ir'^i ki{xi,X2)x{t^2) dxidx2dK2 







Mb{(P; Ki) 




ujo Jk2 {Xl - Ki)(x2 - K.i)yJ{xi - K2){k2 " X2) 



Kl{xi,X2) 



Jui v^(xi - Kl){x2 - Ki) 



X{k2) dK2 



i/(xi - K2){k2 - X2) 



dxidx2 



(4.12) 



for any ki € (— oo,0). Finally, combining (j4.1ip and (j4.12p we obtain for any x 6 C'([0,fo]) and 
any ki G (— oo,0) the equality 



Ki{xi,X2) 



X{k2) dK2 



lo Jui yj{xi - Hi){x2 - Kl) 

In particular, for any A; > and for any ki G (— oo,0), 

^'^Y'" Ki{xi,X2) 



X2 \/{xi - K2)iK2 - X2) 



dxidx2 = 0. 



y(X]"^^"'tl)(x2"^^Kl) 



Rk{xi,X2) dxidx2 = 0, 



(4.13) 



(4.14) 
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where 

Rk{xi,X2):= ^===== 777r=^T (4-15) 

Recall that the Legendre polynomials Pk, k >0, can be generated by the power series expansion, 

00 

(1 - 2u;z + z2)-i/2 = ^ p^(^) / ^ (4.16) 

k=0 

which is convergent for small z. For < X2 < Xi we set Si := (xi + X2)/2 and ,52 := ^XiX2- 
Lemma 4.5. For any A; > and for any < X2 < xi, Rk{xi,X2) = T^s\Pk{si/ 82)- 
Proof. For any given values of xi and X2, < X2 < xi, consider the power series in z, 



I{z) := ^ Rk{xi,X2) z^ 



nk{xi, X2} z^ 
k=0 

There exists < r < 00 sufficiently small such that the power series converges for |2:| < r and 

liz) = I \ / at . 

^ ' Jo t 1- Z{X2 + t{xi - X2)) \ 1-t 



Using the substitution, s = a/ we get 



I{z) 



- ZXl)(l - ZX2) ' 



and by (j4.16p we obtain 



I{z) = TtY^ slPk{si/s2)z^ , 
k=0 

which proves the lemma. □ 
Note that the function 

r \ \ Ki{xi,X2) 

(Xi,X2) ^ Q(Xi,X2,Ki) — 



a/(xi - Ki)(x2 - Kl) 



belongs to uq] x [0, z^i]) in view of Lemma l4.3t and it depends analytically on ki £ {—00, 0). 

Consider the power series expansion 

J 00 

^Qj(xi,X2)V, (4-17) 



i/(xi - /tl)(x2 - Kl) 



k=l 



where (xi,X2) G [^^1,1^0] x [0, i^i] and ki < 0. Now ()4.16p implies Qj{xi,X2) = — S2 ^ Pj-i{si/s2) 
for any j > 1. 

Using Lemma |4.5| (|4.14p and (j4.17p we obtain that for any k,j > 0, 



Jui 



kl{xi,X2)sl'^^ Pk{si/ S2)Pj{si/ S2) dxidX2 = 0. (4.18) 
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Let k and m be non-negative integers such that 2k < m and let d be the integer part of m/2. 
We have the following relation due to Adams (see [I], [16\ Chap. XV, Legendre functions, 
Miscellaneous Examples, Ex. 11]), 

k d 
r=0 ■r=0 

where for any < r < d, 

^ _ Ak_rArA.^-k~r 2m - 4r + 1 
'~ Am-r ■ 2m - 2r + 1 ' 

with 

^k-=\ 1, A; = 0, 

Aa; = 0, A;<-1. 

Hence, for any given m > we obtain a {d+ 1) x (d+ 1) matrix (c™^)^ which is triangular (all 
the elements over the diagonal vanish) and with non-vanishing diagonal elements. This together 
with (I4.18P (take m = j + k) implies that for any m > 0, < 2r < m, 

/ Ki{xi,X2)s^Pm~2r{si/ 82) dxidX2 = 0. 
J VI 

On the other hand, for any m > the monomial can be written as a linear combination of 
the Legendre polynomials Pm-2r{z)^ < 2r < m, and we get 

/ / Kiixi,X2)sT~^''sl'' dxidx2 = 0. (4.19) 

Jo J VI 

Consider the set of monomials = {s^'"^^ : r, m G Z, < 2r < m}. Obviously M is closed 
under multiplication, 1 G Ai, and it separates the points (xi, X2) of the compact [vi, vq] x [0, ui], 
since si, S2 ^ Ai and < X2 < xi are the unique solutions of — 2six + S2 = 0. The Stone- 
Weierstrass theorem implies that the vector space Span(A^) of all finite linear combinations of 
monomials of Ai is dense in C( [1^1,^0] x [0,z/i]). Choose Tp G C( [1^1,1/0] x [0, z^i]). Then for any 
e > there is P G Span(A^) such that 

\\P - '^\\ci[vuvo]x[0,vi]) < £• 

Now ()4.19p implies 

VI r-vo 

I Ki{xi,X2)lp{xi,X2)dXidX2 
Jvi 
VI ruo 

/ |i^i(xi,X2)||V'(2;i,a;2) - -P(xi,X2)|(ixi(iX2 < e||-^l||Ll((i.i,i.o)x(0,;.i)) • 
J VI 



< 



Hence, 

fVi rvo 

/ i^i(xi,X2)V'(2;i,X2)(iXi(iX2 = 
Jui 
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for any ■0 S C([z^i,i^o] ^ [Oj'^i]) which imphes Ki = on that compact. In particular, /C = 0, 
and hence K = Q. This completes the proof when = 1. 

Now, consider the case when //(^) = {'k'^ {£),ng)~^ . Assume that ^ € A/j where A/^ is a 
Liouville torus in and h = (/ii,/i2) are the values of the integrals Ii and I2 on A/^. Let 
A/j C C/i- Using the mapping (j3.3ip . we introduce coordinates {(^1,6*2)} on the "half tori 
Ti^l = and = ri,_i(4) of T^^^ as weh as on rf^^ = r_nK) and rfj = r_i,_i(4) 

of . Similarly, we parametrize the Liouville tori in U2 ■ 

Lemma 4.6. In coordinates {(^1,^2)} on A^ C J^h, /"(O = (^^(Oj'^s)^^ ^-^ given by 



,ie,,02) = ^ll^^^^^^^ (4.20) 
where ki = hi + hi and K2 = hih2- 

Proof of Lemma Fix h = {hi, /12) so that A/j C Th- It follows from ()3.5p that 

1 a 

On the other hand, the third equation in (|3.13p shows that 

P3 = -\l - hii^s + h2 = -\/ (i^i - '^3){k2 - 1^3)- 



Hence, (7r+(^),ng) = ^(ki - J^3)(k2 - i^3)/\/Tl^. □ 



Remark 4.7. T/ie statement of Lemma \4-6\ holds also for any ^ T not necessarily in Th- 

Note that the denominator in (|4.20p is a positive constant on and the numerator is indepen- 
dent of hi and /12 and does not vanish. The relation (14. 3p with 



k{Qi,d2) = /C(^i, ^i)(^i(0i) - ^2(^2))v/(V'l(^l)-^3)('^l(^l)-l^3). 

implies that the expression (j4.5p vanishes. In particular, (j4.7p and ()4.8p hold. Finally, arguing 
in the same way as in the case [i=\ one concludes that = 0. □ 



5 Non-degeneracy of the frequency map 

In this section we investigate the non-degeneracy of the frequency map of Liouville billiard tables 
of classical type. 

Theorem 5.1. Let {X,g) be an analytic dimensional Liouville billiard table of classical type. 
Suppose that there is at least one non-periodic geodesic on P. Then the frequency map is non- 
degenerate in the union Ui U U2 corresponding to the boundary cases (A) and (B). 
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Proof. As in Sect. l3.3l we introduce coordinates {(^i, 6*2, ^3;Pi,P2iP3)} on the cotangent bundle 
T*C, where pi, p2, Ps are the conjugate variables to 9i, 62, and ^3. Solving the system of 
equations (|3.12|) with respect to pf, p^; Ph where H = 1, Ii = hi, and I2 = /12 are given 
values of the integrals, we get 

Pi = ^1- ^iVi + h2 
' pI = -{'A-hW2 + h2) (5-1) 
^ pI = "fl- ^iV^s + h2 ■ 

In particular, it follows from (j5.ip that the invariant set 

Th:={H = l,h = hij2 = h2}ciT*C (5.2) 

is non-empty if and only if the quadratic polynomial P{t) = — hit + /i^ has real roots ki < K2 
(i.e., V = hi — 4/i2 > 0). As in Sect. H] we obtain four cases related to the position of the roots 
Ki and K2 with respect to the constants < 1^2 = < ui < vq, namely, 

(A) I's < Ki < 1^2 = and < K2 < 1^1; 

(B) 1^3 < Ki < and z^i < ^2 < fo; 

(C) < Ki < K2 < i/i; 

(D) < Ki < z^i and i^i < K2 < vq- 

Recall that = min(/?3, V2 = max(/73 = min(/?2 = 0, 1^1 = max(/92 = min(/?i, and uq = maxy?i. 
In what follows we consider ki and K2 as new parameters (constants of motion jl that parametrize 
the invariant set (j5.2p . 



We first consider the case {A) where < ki < 1^2 = and {) = V2 < H2 < vi- It follows from 
5.ip that the impulses are real-valued if and only if 



vi < 'fi{Oi) < 
< ip2{92) < m 

V-i < ^3(93) < Ki 



(5.3) 



Hence, the projection of the invariant set (|5.2p onto the base C is described by the following 
inequalities: 

< 6*1 < ; 

-/2(^2) < ^2 < /2('^2) or - 72(^2) + Y < ^2 < /2(/^2) + Y ; 

/3('Cl)<^3<iV or -N <e3<-f3{^i), 

where /2 is the inverse of '/72|[o,t^2/4] ^"^^ /s inverse of V53|[o,Af]- These inequalities give four 

rectangular boxes in C that project onto an unique set in C via the projection (13. 3p . Consider, 



^hi = Ki + K2 and h2 = «;iK2 



25 



for example, the rectangular box Bfi given by 



< 6*1 < ; 

-/2(/^2) <^2 </2(k2); (5.4) 
/3('^l) <^3 <iV. 



For any given 6 S we obtain from (jS.ip that 

Pi{0) = el^/{^l{el)-Kl){^l{el)-K2) 



P2{0) = e2\/ (^2(6*2) - Ki)(k2 - ip2{02)) 



where = ±1. Then the mapping r-|_ : T*C, 

{01,62,63) ^ {61,92,63; pi{6),p2{6),p3{6)), 

where ei = 1, €2 = ±1, and €3 = ±1, parametrizes one of the two connected components 
of the subset Th := {H = = /ii,/2 = /12} C T*X. Assume that the strict inequalities 
1^3 < Ki < z^2 = and < K2 < I'l hold. 

Remark 5.2. This component is diffeomorphic to x [0, 1] and its intersection with the bound- 
ary T*X\y ofT*X has two components which can be identified with the two components of the 
image of the slice 

{0<ei< LOi, -f2{K2) <62< /2(k2), 63 = N} 

of Bh with respect to r_|_ with £3 = 1 and €3 = —1 respectively. In particular, the impulse p3 takes 
constant values of different sign on them. Moreover, the reflection map r : T*X\-p T*X\y is 
given by 

{6i,62;pi,P2,P3) ^ {6i,62;pi,P2, -ps) ■ 

Hence, the reflection map interchanges these two components, and by Lemma \ 7.4\ {c), m = 1 
(cf. Remark \2. Similarly, we get m = 1 in the case {B). 



Now we compute the generalized actions of the billiard flow corresponding to Th (see (17. Sp . 
Appendix), 

Ji{ki,K2) = ttz I V {^i{6i) - Ki){ipi{6i) - K2) ddi 

y/{xi - Ki){xi - K2)pi{xi)dxi , (5.5) 







27r 

2 



J2(ki,K2) = Z V {'P2{62) - Ki){k2 - IP2{62)) d62 

'-/2(k2) 



TT 



IT 



V {X2 - Ki){k2 - X2) P2{X2) dX2 , (5.6) 
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vr 



J3{kI,K2) = Z V (^1 - <^3(6'3))('«2 - ^sids)) dO; 

'/3(«i) 



V (ki - X3)(k2 - X3) /03(X3) dX3 , (5.7) 

'1/3 

where 

Pi{xi) ■■= f[{xi) > 0, P2{x2) ■= f2{x2) > and psix^) := -/3(x3) > 

are analytic functions in the intervals {ui,uq), (0,z/i) and (^3,0), respectively, and /i and /2 
satisfy iK27\\ and ([329]). Notice that the functions and in (l3:27|) and (K29\\ are smooth 
and even analytic in a neighborhood of 0, and f^^x^) = \/— ^-^3(\/— X3), where -F3 is analytic in 
a neighborhood of [2^3,0]. By the assumption (^5), paix^) is smooth at X3 = 1^3. In particular, 
we obtain 

Remark 5.3. The functions pi, p2 and are analytic in the intervals {vi,uq), (0, 1^1) and 
(f3,0), respectively, and 



P'^i^^) = \/xi~ui as xi ^ ui + and pi{xi) = as xi ^ uq - 

P2{x2) = as X2 ^ + and P2{x2) = '^'^^fj^^'J^^ - as X2^ i^i-Q and 

/'3(X3) = ^2^^ as X3-0-0, 

where and are analytic in a neighborhood of 0, and G3 is analytic in a neighborhood of 
[1/3,0]. Moreover, G^iO) > and by ([3:28]) and (l33U]) we have 



G+(o) = V2(^;'(o)-i , Gr(o) = -y^vn^OT" 



G+(0) = VMW^, ^2(0) = -V-2v9'2'(^2/4)-i . 
As a corollary we obtain 

Lemma 5.4. The functions Ji, J2, and J3 are analytic in (^1,^2) S (^3,0) x (0, z^i) 



Proof of Lemma \5.4\ The function Ji is obviously analytic in that domain. Fix a G (0,1^1) and 
take < 6 <^ 1 such that P2{z) is holomorphic in the disc lD)25(a) := {\z — a\ < 26} C C. Then 
write 

ra—S 

J2(ki,K2)=/ - 3^2 /(a:^2, Kl) dX2 + / VK^^^f{x2,Ki)dx2, 

Jo Ja-S 

where ^ 

/(x2,Kl) = -VX2 - ^2(2:2) 

vr 

is analytic in (0,1^1) x (2^3,0). Then the first integral defines an analytic function in (^1,^2) G 
(f3, 0) X (a — 5/2, a + 5/2). Consider now the second one. We expand f{x2, ki) in Taylor series 
with respect to X2 at X2 = K2- Then integrating with respect to X2 and using Cauchy inequalities 
for ^(k2j i^-i), where (k2, ki) € B5/2(o) x (1/3 + 5, —6), we obtain that the second integral defines 

an analytic function in (^1,^2) G (1^3 + 6, —6) x 05/2(0)- In the same way we prove that J3 is 
analytic in (2^3, 0) x (0,1^1). □ 
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In order to obtain suitable formulas for the frequencies of the billiard ball map we proceed 
as in the Appendix. Denote by 'H{Ji, Ji, J3) the Hamiltonian of the billiard flow expressed in 
the corresponding action-angle coordinates. Then for any ki and K2 such that z^3 < ki < and 
< /t2 < 1^1, one has 

W(Ji(ki,K2), Ji(k;i,K2),^3(/^1,/^2)) = 1- (5.8) 



Differentiating (jS.Sp with respect to ki and K2 we get that the frequencies VLi and 1^2 of the 
billiard ball map satisfy 



dJi dJ2 dJ-j 

Oki dni dK\ 

dJi dJ2 dJ-j 

dK2 dK2 dK2 



2tt 







and therefore (cf. formula (17. 7p in the Appendix) 



9Ji 


8J2 " 




' ni ' 




dJ'i ' 


9k 1 






= -27r 


(9k 1 


dJi 


dJ2 




. ^2 . 


dJz 


- dK2 


dn2 - 






. dK2 . 



(5.9) 



The latter relation and the formulas for the actions (j5.5p - (j5.7p lead to the following formulas for 
the frequencies 



«;2j = vr ^ and i,l2[Ki, K2) = ■ 



where 



and 



A{ki,K2) 
B{ki,K2) 

D{k,i, K2) 



D{ki,K2) D{ki,K2) 

(x2 - 3^3)^2(3^2)^3(3:3) dx2 dX3 

Ki Jo V (X2 - Ki){k2 - X2){ki - X3){k2 - X3) ' 
"1 (Xi - X3)pi(xi)p3(x3) dxi dX3 

U3 Jui (xx - Kx){xx - K2){ki - X3){k2 - X3) 

{Xi - X2)pi{xi)p2{x2) dxi dx2 



(5.10) 



Jux \/ {Xi - Ki){xi - K2){X2 - Ki){k2 - X2) 

It follows from Lemma [5.41 that A, B and D are analytic functions in (ki, K2) G (z^3, 0) x (0, vi). 
Moreover, D 7^ in that domain, which implies that Vli and ^2 are analytic in {ki,K2) € 
(1/3,0) X (0,1/1). 

Denote by J the Jacobian of the frequency map (ki, K2) 1— > {^i{ki^ ^2), f^i('^i; '^2)), 



J{ki, K2) :-- 



d{ni,n2) 



d{Kl,K2) 



A^,D-AD^, A,,D-AD^ 



K2 
K.2 ■ 



(5.11) 



Since J{ki,K2) is analytic in (^1,^2) G (^3,0) x (0,i/i), either J{ki,K2) / in an open dense 
subset of (1/3, 0) X (0, ui) or 



J{ki,K2)=0 for any (ki, K2) G (1/3, 0) x (0, i^i) , 



(5.12) 



We are going to compute the limit of J^{ki,K2) as ki ^ U3 + 0. To do this we will need the 
following auxiliary Lemma. 
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Lemma 5.5. Let f{x,K) be a function on (a, 6) x (a, 6) such that f and its partial derivatives 
fx, fn o-nd fxK exist and are continuous and bounded on (a, 6) x (a, 6). Consider the function 

:= r '^^ dx . Then 

Ja ^JK-X 

(a) F{k) = 2f{a,K)^/J^^ + 0{\K-a\^/^) 
y/H — a 

where the estimates above are uniform in k £ (a, b) . 
Proof of Lemma 15.51 An integration by parts leads to 

F{k) = —- / /(x, k) d\J K — X = 2f{a, k)V k — a + 2 / fx{x, k)\J k — x dx (5.13) 

^ J a J a 

that together with the boundedness of fx proves (o). Differentiating (j5.13p with respect to n 
and using the boundedness of fx, fn, and fxn, we prove (6). □ 



The expression for A(ki, K2) can be rewritten in the form 

A{ki,K2) = 

J U: 

where 



f{x3,Ki;K2) 

dX3 



/(x3,ki;k2) := 



U3 Vi^i- X3 

P3ix3) f'^ ix2 - X3)p2{x2) dx2 



X3 Jq ^ (x2 - Ki)(k2 - X2) 



For any given K2 S (0, z^i) the functions /(x3,ki;k2) and ^'^^^g^T'''^'' satisfy the conditions of 
Lemma [531 (with x = 2:3, k = Ati, a = ^3 < < 6 < 0) in view of Remark 15.31 Applying the 
Lemma we get 



\( \ lo ^3(1^3) f WX2 - J^3 P2{X2) dx2\ . , , . . 

A(ki,K2) = 2 = / , VKi - 1/3 + o(Vki - 1^3) (5.14) 



dA{Kx,K2) ( p?,(v?^ f WX2 - 1'3P2{X2) dx2\ / . , ,^ , , , 

7^ = \ I = , /Vki - Z^3 +o(l/V'«l - 1^3) 

dUi \ ^K2 - ^3 Jo Vl^2 - X2 'I 



(5.15) 



and 



dK2 dK2 V ^JK2 - V3 Jo V ^^2 " X2 

as Ki — > 1/3 + 0. In the same way one obtains 



dA{Ki,K2) d ( P3{U3) f'^^^/^^^^P2{x2)dX2\ 

73 = 2^-—^ / , V'^i - i^3 + o(V'«i - 1^3) (5.16) 



^(ki,K2) = 2 = / , VKi - 1^3 + o(V'^i - 1^3) (5.17) 

V ^K2 - V3 Jui Vxi - K2 ) 



dB{Ki, K2) 



) fPsij^i) f Wxi - U3pi{xi)dxi \ / 

-=[^^^= , ] /VKl-U3+o{l^/Kl-V3) (5.18) 

V 7^2 -1/3^1/1 VXI-K2 )l 
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and 

dB{Ki 



.(ki,K2 ^ d p-i{u-i) f o^xi - U3pi{xi)dxi\ 

-t; = 2— / , V^i -1^3+ o{Vki - 1/3) (5. 



19) 

as Ki — > + 0. Note also that for any K2 G (0, z^i), D[ki, K2) is a continuous (even real-analytic) 
function with respect to ki on the whole interval (— oo,0). 

Consider the limit 6{k2) := lim tt^'^ D"^ J' {ki , K2) for K2 G (0,z^i). It follows from (I5.1ip 

and dSHD-dSlSI) that 



-S«,^«,)+o(l) 



rUQ \/x\—vz pi(xi) dxi 



+ 0(1) 



as Ki — > f3 + 0. Hence, 



/ rug ^/xi~-u-j pi{xi) dxi 



\/k2—X2 



^K2 - V3' V Jo -y/K2 — 2^2 ^ C^«^2 I Va^2-i^3 P2(a:2) Q!a;2 

\ -J 



Suppose that (|5.12p holds. Then 5{k2) = for any K2 G (0,i^i) and it follows from (j5.20p that 
there is a constant C 7^ such that 



^/ir^^Pl(a;i)(ixi _ ^ ^X2 - V3P2{X2) dx2 22^) 



2^1 



V^^l - K2 Jo ^JK2 - X2 



for any K2 G (0, i^i^ 



Lemma 5.6. Let {X,g) be a Liouville billiard table of classical type. Then the geodesic flow of 
the restriction I = g\-p of the Riemannian metric g to the boundary T is completely integrable. A 
functionally independent with I integral of the geodesic flow of I is given by the restriction / = /2|r 
of I2 to r and the level set {/ = 1, 1 = k} is non empty if and only if n ^ [0, vq] . In action-angle 
coordinates the rotation function corresponding to the Liouville torus := {I = 1,1 = for 
K £ (0, vi) is 

p{k) = 2 n V^i ~ V3pi{xi)dxi I p Vx2 - 1^3 P2{x2) dx2 



\JX\ - K I Jq ^Jk - X2 



Proof of Lemma 15.61 It follows from the construction of the Liouville billiard tables that the 
mapping a\f2 : — > P is a double branched covering of the boundary L, where 



= {(01 (modwi),02 (modu;2),03 = iV)} C C. 



^This set has two connected components that correspond to two Liouvihe tori with the same rotation function. 
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In the coordinates {(^1,^2)} on we get the following expressions for the metric / = {a\j2^)*l 
and the integral / = {cr\j2 )*I 



Applying the Legendre transformation corresponding to / we obtain the following system of 
equations for the level set := {/ = 1, 1 = k}, 



1 / pI 

— 099 Vwi — 



I = [ip2 h<^l 

that leads to the following expression of the impulses on T^, 

piiOif = {M0i)-^3){MOi)-f^)>O (5.23) 

P2{02f = {M02) - 1^3) - MO2)) > 0. (5.24) 

In particular, 7^ if and only if k G [0, vq]. Hence, the projection of into the base is 
given by the union of the sets 

K ■■= {(.01,02) : 0<9l<UJi, -f2{K) <92< f2{K)} 

and 

K ■■= {{0l,02) : 0<ei<OJi, -f2{K) + 002/2 < 02 < f2{K) + OJ2/2} . 

As the sets A'^ and A'^ have the same image under the projection cr\j2^ : — > F we restrict our 



attention only to the set A'^. It follows from (I5.23j) - (l5.24p that the mapping r+ : A'^ -^T*T%, 

{OiM ^ {61,92; ViMOi) - 1^3) MOi) - K),±^{ip2{92) - us) {k - 992(^2)) ), 

parametrizes one of the two connected components of the set = {/ = 1, / = k} C T*X. By 
Liouville- Arnold formula we get the following formulas for the corresponding actions 



vr 



Ji(k) = - / y/{^^{e^)-u3){^i{ei)-^) d9i 

V {xi - 1/3) {xi - k)pi(xi) dxi (5.25) 







J2{t^) = - r^'^^ ViM02)-U3){fi-M02))d92 



vr 



a/ (x2 - V3) - X2)p2ix2) dx2 (5.26) 
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In the corresponding action-angle coordinates the Hamiltonian L becomes L = L^{Ji, J2), where 
is smooth, and the frequency vector to of the invariant torus {Ji = ci, J2 = C2} is w = 

— ^§j^(ci, C2), ^7^(ci, £2)^ . Then, differentiating the relation 

L\Ji{k),J2{k)) = 1 

with respect to k G (0, vi) we get (j5.22p . □ 
We need the following technical Lemma. 

Lemma 5.7. Let m< < M be real constants, Fi £ C^{[0,M]), and F2 £ Ci([m, 0]). Then 



and 



[ dt = -2Fi(0) log + 0(1) (5.27) 

Jo vt^t — a 

r^i^^Ldt = 2F2(0)log^ + O(l) (5.28) 



as a —>■ — 0. 
Proof. We have 

rVM 



r F ( /f) M 1 

/ /)l_ rft = 2Fi(0) / _ du + 0(1) = -2Fi(0) log ^ + 0(1). 

Jo \Jtyt — a Jo \/u^ — a 

The proof of (j5.28p is similar and we omit it. □ 
Lemma [5. 71 can be applied to the two integrals in ()5.22|) using Remark 1 5. 31 In this way we obtain 



\ , = -2Gf (O)Vi^i - i^slog Vz^i - K + 0(1) 

Jv\ \JX\ — K, 



and 



/ , = 2G2 (O)Vi^i - z^alog V^^i - K + 0(1). 

On the other hand. Remark 15.31 and assumption (^43), (2), in Sect. 13.11 imply 

0^(0) = -^-2(^'2'(a;2/4)-i = -^2^'({0)-^ = -G+(0), 
and by (j5.22p we obtain 

p{k) ^ 2 as K — > z^i — . 

As by (|5.2ip . p = const we conclude that p = 2 on the interval (0, z^i). The latter implies that 
all the geodesies of F lying on a torus with k £ (0, ui) (see Lemma l5.6p are periodic. Using 
the analyticity of the billiard table and considering the Poincare map in a tubular neighborhood 
of the "hyperbolic" level set = 1, / = z^i} we obtain that any geodesies of T corresponding to 
some At G [z^i, 1^2) is periodic as well. As the level sets {/ = 1, / = 0} and {1 = 1,1 = fg} consists 
of periodic geodesies we see that all the geodesies on T are periodic. Hence, the assumption that 
the Jacobian J of the frequency map vanishes in an open subset of {ki,K2) £ (1^3,0) x (0,z^i) 
implies that all the geodesies of F are periodic. The case (B) can be studied by the same 
argument. □ 



32 



6 Proof of Theorem [T] and Theorem [3] 



In this section we prove Theorem [3] and Theorem [T] formulated in the introduction. Let {X^g) 
be a 3-dimensional analytic Liouville billiard table of classical type such that F := dX admits 
at least one non closed geodesic. 

We will prove a more general result than Theorem [3] which requires only finite smoothness 
of Kt. Namely, fix d > 1/2 and £ > 4 [2d] + 11, where [2(i] is the entire part of 2d and d is the 
exponent in (Hi). Denote by C^(r,M) the corresponding class of Holder continuous functions. 

Theorem 6.1. Let [0,1] 3 t ^ Kt he a continuous curve in C^(r,M) and suppose that it 
satisfies {Hi) and {H2), where I and d are fixed as above. If Kq and Ki are invariant with 
respect to the group of symmetries G, then Kq = Ki . 

Proof. Given a > and r > 2 we denote by the set of all frequencies {0.1,0.2) G satisfying 
the Diophantine condition 

For any (A:i,A;2,/c3) G Z3,(A:i,A;2) ^ (0,0) : \Oiki + 02k2 + h\ > j—. — ^ . 

(jKil + |fe2|) 

Note that the set O'^ := Ua>oO.'^ is of full Lebegues measure in for any r > 2 fixed (cf. 
[HI Proposition 9.9]). Then it follows from Theorem 15.11 that the subset of Ui U U2 filled by 
invariant tori A with frequencies in 0^ is dense in Ui L) U2. Take 0<r — 2^1so that 
i > {[2d] + l)(r + 2) + 7. Then we apply [12, Theorem 1.1] for any A in that family. By Remark 
15.21 we have 

7^Xo,/.(A) = TIk.A^) (6.1) 

for any t G [0,1] and for any torus A with frequency in Q'^, where fi = (7r"*"(^), n^)"^. By 
continuity we obtain (j6.ip for any Liouville torus A lying in the part U1UU2 of B*T corresponding 
to the boundary cases. Finally, Theorem 16. II follows from ()6.ip and Theorem 14.11 □ 

Proof of Theorem [7]. Let {X, g) be a 3-dimensional analytic Liouville billiard table of classical 
type and let = 1 or /i = {tt^ {£^),ng)^^ . Assume that K G C(F,R) is invariant with respect 
to the group of symmetries G = (Z/2Z)^ of F and let the mean value of /i • X on any periodic 
orbit of the billiard ball map be zero. It follows from Theorem 1 5 . 1 1 that the set filled by Liouville 
tori A of the billiard ball map with frequency vectors := {0,1,0,2) G Q x Q is dense in the 
part of B*T corresponding to boundary cases. Let A be such a rational torus. In action-angle 
coordinates, A = R^/Z^. There exists G N and two relatively prime numbers p,q £ 7j such 

that O = (^jj, (modZ^). Hence, there is an affine change of coordinates on R^/Z^ such that 

O = {1/N,0) (modZ^). Denote, 

V := {{x, y) : < x < l/N,0 < y < 1}. 

Using the invariance of A and of the Leray form on A with respect to we obtain, 

N N 

f{f^-K)X = ^Y. [ (^*)'(/^ ■K)X = [ ■K))X = (6.2) 

J A fc=l-^^ -^^ k=l 

as by assumption the mean ' ^) vanishes. Using the density of rational tori A in 

boundary cases, equality (|6.2p . and Theorem 14.11 we see that K = 0. □ 
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7 Appendix: Frequencies of integrable billiard tables 



In this appendix we collect the necessary facts used for the computation of the frequency map 
in Sect. [SJ Our main task is to derive formula (j7.9|) for the frequencies of the billiard ball map. 

Let {X,g), n = dimX > 2, be a billiard table with non-empty locally convex boundary F. 
Consider the reflection map at the boundary, 

p:TX\r^TX\r, ^ ^ i - 2g{i,ng)ng, (7.1) 

where rX|r := G TX : 7r(^) G F} is the restriction of the tangent bundle to F, vr : TX X 
is the natural projection onto the base, and Ug is the inward unit normal to the boundary. The 
restriction p is an involution on TX\y the set of fixed point of which coincides with TV C TX\t- 
Note that p preserves the values of of the Hamiltonian Hg{£) := ^g{£,,0 ^^'^ when restricted 
to the unit spherical bundle SgX\r := G TX\r : = 1} it coincides with the mapping 

r : S ^ S considered in Sect. [2] if we identify vectors and covectors with the help of the Legendre 
transform, 

FLg : TX ^ T*X, ^ ^ g{C, •) . 

More generally, the notions and mappings considered in Sect. [2] have their analogs on TX via 
the Legendre transform. 

Denote by ag the Liouville 1-form on TX given by ag{v)(-) := g(v,dv'n'{-)) where v G TX 
and (•) stands for an arbitrary element of T^{TX). Note that the differential ujg := dag of the 
1-form Og corresponds to the symplectic form dp A dx on the cotangent bundle T*X via the 
Legendre transform. 

Lemma 7.1. The reflection map p : TX\y TX\-p satisfies the following properties: 

(a) the reflection p preserves the restriction of the Liouville form Og to TX\y; 

(b) the reflection p preserves the values of the Hamiltonian Hg{S,) = ^^(CiO;' 

(c) in the case when {X, g) is a Liouville billiard table the reflection p corresponding to the 
Riemannian metric g preserves the values of the pairwise commuting integrals Ik {k = 1,2) 
of the billiard flow (cf. Proposition VS. 'J\) . 

Proof of Lemma \7.1\ (a) Let t ^ v{t) be a smooth curve in TX\r defined in an open neighbor- 
hood of t = such that v{0) = v £ TxX, x G F, and ^(0) = H G Ty{TX\r). One has 

ag{p{v)){d^,p{E)) = g{p{v),dp(^^)TTodvp{E)) = g(^p{v),^Tr{p{v{t))\t=o] 

= 9(v - 2g{v,ng)ng, ^7r(w(t))|t=o) = 9(v, ^vr(?;(t))|t=o^ 
= c,giv){E) (7.2) 

where we have used that tto p = -k and that ^7r(ti(t))|j=o G T^T is orthogonal to Ug. This proves 
statement (a). The proof of (6) is straightforward and we omit it. Statement (c) was established 
in the proof of Proposition 13.31 □ 

Now we will describe a special variant of the symplectic gluing procedure introduced by 
Lazutkin in [H § 4]. The main idea is to identify parts of the boundary d{TX) of the configuration 
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TX\TT B < 



space TX of the billiard flow in order to "eliminate" the reflections and obtain a new "glued" 
configuration space together with a smooth billiard flow on it. Note that the glued configurations 
space becomes a smooth symplectic manifold so that the billiard flow is a smooth Hamiltonian 
system on it. Divide the boundary of TX into three parts 

d{TX) = TX\r = T'X\r U T+X\r U TV 

where T^X\r := {£, G TXr ■ ^g{S,, ng) > 0} and TV is assumed naturally embedded into TX\y. 
Note that 

p\t-x ■■ T-X\r ^ T+X\r (7.3) 

is a diffeomorphism and the elements of TV C TX|r are fixed points of p. Now, using ()7.3p we 
identify the points ^~ G T~ X and p{£,~) G T^X of the boundary of TX \ TV and obtain a new 

glued space TX that we supply with the factor topology so that the projection vTp : TX \ TV 
- — p 
TX , 

' e if ^ ^ diTX) 

{c,p{m if eer-xir , 

^ {p-HO,^} if eer+Xlr 
is continuous. 

Definition 7.2. The billiard flow of {X,g) is called completely integrable if there exist n 
functionally independent integrals Qi,--.,Qn = Hg £ C°°{TX,M) of the billiard flow such that 
yi<k,l<n, {Qk,Qi} = 0, and VI < A; < n Ve G TX\r, QkipiO) = QkiO- 

Assume that the billiard flow on TX is completely integrable. Denote by Xg the Hamiltonian 
vector field on TX with Hamiltonian Hg. The following Proposition follows from Lemma l7.ll 
(a), (6), and is a special case of the symplectic gluing developed in [8, § 4]. 

Proposition 7.3. There exists a smooth differentiable structure on TX^ , a symplectic form uig 
on TX^ , and functions Qk G C°^{TX^ {1 < k < n), such that the projection 

TTp:TX\TT ^ fx^ (7.4) 

is smooth, 7r*(u)g) = iOg, and ■K*p{Qk) = Qk for any 1 < k < n. In particular, the Hamiltonian 
vector field Xg corresponding to Hg := Qn is completely integrable in TX^ and (7rp)*(Xg) = Xg. 

Denote, 

r := TTp{T^X\r) C TX^ 

Note that T is a disjoint union of connected non-intersecting embedded hypersurfaces in TX^ 
that are transversal to the Hamiltonian vector field Xg. 

Let c = (ci, Cn) be a regular value of the "momentum" map 

M-.fx'^R^, (Qi(e),...,Qn(e)) 

and let Tc be a connected component of the level set M~^{c). The compactness of X implies 
that Tc is compact. By the Liouville- Arnold theorem Tc is diffeomorphic to the n dimensional 
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torus T" and one can introduce action-angle coordinates in a tubular neighborhood of in 



TX'' (see 12 1). Assume that 



is a non-empty compact set. In this case we will call Tc glued Liouville torus. As Xg is tangent 
to Tc and transversal to T the submanifolds Tc and T intersect transversally. Hence, TcCiT 
is a disjoint union of finitely many compact embedded submanifolds in Tc- Denote by m > 1 
the number of the connected components of Tc H T. The proof of the following Lemma is 
straightforward and we omit it. 

Lemma 7.4. 

(a) The connected components ofTcnT are diffeomorphic to T^~^ ; 

(b) The closure of any of the connected components ofTc\T is diffeomorphic to [0,1] x T"^^. 
The n-torus Tc is obtained by a "cyclic" gluing together of all m > 1 copies of [0, 1] x T"^^ 
along their boundaries; 

(c) LetSc be a connected component ofTcCiT and let Ac = p+(7r~^(5c)) wherep^ : TX\y TT 
denotes the orthogonal projection ^ g{ng, £,) Ug onto TT. Then the number m > 1 of 
the connected components of TcCiT is the minimal power of the billiard ball map B that 
leaves Ac invariant, i.e., B^{Ac) = Aclll 

Choose a component Sc of Tc n T and a basis of cycles 71, ...,jn-i of its homology group as 
well as a transversal cycle 7^ in Tc so that 71, 7^ is a basis of the homology group of Tc. Let 
{ Ji, Jn] 6*1 (mod 27r), 6'„(mod 27r)} be action-angle coordinates in a tubular neighborhood of 
the glued Liouville torus Tc that corresponds the the cycles 71, ...,7n, i-e., VI < A; < n, 

where Ug := {'^p)*{a:g) is the push-forward of the Liouville form Og onto TX^ . In the action-angle 
coordinates, 

X,=,,(i)J- + ... + ,„(,7)4-, 

where _ 

rik{Ji, J„) := —^{Ji, Jn), I <k <n. (7.5) 
dJk 

It follows from the choice of the cycles 71, ...,jn that Sc is a section of the bundle, 

T"-i X T ^ T^-\ {§1, 4) ^ {9i, . 

As Sc is transversal to Xg one concludes that 7/„(J) 7^ 0. It follows from our construction that 
the billiard ball map B"^ is conjugated to the following diffeomorphism of the n — 1-dimensional 
torus {(0i(mod 27r), ...,6'„_i(mod27r))}, 

Ok^ 9k + 27r^^^^, l<k<n-l. 
Vn{J) - - 



^Note that we identify vectors and covectors via the Riemannian metric g. 
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Parameterizing the glued Liouville tori with fixed energy {Hg = 1} by the values of the integrals 
Q = {Qii •••) Qn-i) we obtain the following mapping for the frequencies of B"^, 

(r2fc((5i,...,(5„_i))i<fc<„_i := 27r ( ^''^'^j^^' ...,Qn-iA)) \ ^^^^ 

V»?n(j(Qi,...,Q„-i,i)) 

where ??fc(Ji, ■■■,Jn) is defined by (j7.5p . Finally, by partial differentiation of the identity, 

ng{JiQl,...,Qn~l,l)) = 1, 

one gets that the frequency vector := (ili, satisfies the linear relation 

AO = -27rb (7.7) 

where A(Q) := 1)) , b(Q) := 1), gg^CQ, 1))^, and VI < A: < n, 

hi"" <^-«> 

where 7^ is the connected component of vr~^(7fc) lying in T+X|r and Q := (Qi, ...,(5„_i). The 
functions Jfc (1 < ^ < ra) will be called generalized actions of the billiard flow. Using that rjn 7^ 
one can prove that A.[Q) is non-degenerate. Hence, 

Vt{Q) = -2^A{Q)-^h{Q), (7.9) 

where Qi, Qn-i are the integrals of the billiard flow in a tubular neighborhood of the invariant 
set 7r~^ (Tc) of the billiard flow. 
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